DNS - implementing TCP #486

McStork · 2015-12-09T13:30:26Z

Thanks @andrewkroh for the rebase.

elasticsearch-release · 2015-12-09T13:32:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

andrewkroh · 2015-12-09T14:10:57Z

packetbeat/protos/dns/dns.go

+		}
+	}
+
+	payload := make([]byte, 0)


The preferred way to declare and empty slice is var payload []byte because it avoids an allocation if the slice is never appended.

andrewkroh · 2015-12-09T14:44:40Z

@McStork Thanks for the great work! I left a few comments (all minor).

McStork · 2015-12-09T14:46:50Z

Thanks for the review. Will update soon.

McStork · 2015-12-10T17:40:02Z

Gaps are better managed now. However I wounder about the PrepareForNewMessage method because the stream.data is not reset, only stream.message is. But if it was reset, we would not be able to get the ID field of the original stream from the reverse stream.

I will have to read more of the tcp.go file to make sure I understand how all methods of an app layer work together ^^'.

McStork · 2015-12-14T09:01:02Z

@andrewkroh As you suggested in the review, I added Notes for when a response cannot be decoded in a TCP stream (Fin and Gap). That's a partial implementation of the TODO list point 'Publish a message when packets are received that cannot be decoded.'.
In order to publish notes for a Query that cannot be decoded, we would need a different way to build the transaction's hash. To publish such Notes would be really useful as you said in the ParseUdp method:

// This means that malformed requests or responses are being sent or
// that someone is attempting to the DNS port for non-DNS traffic. Both
// are issues that a monitoring system should report.

@andrewkroh Do you want to do more review ? Can we expect a merge soon ? =)
It would be nice if I could start working on EDNS/DNSSEC this week.

tsg · 2015-12-15T09:44:48Z

LGTM, yes, I think we can merge it today, but I'd like to wait for @andrewkroh to have one final look.

* Implement DNS over TCP * Publish Notes when a response fails to decode in Gap and Fin

andrewkroh · 2015-12-15T15:53:08Z

packetbeat/protos/dns/dns.go

+
+	var payload []byte
+
+	// Offset is critical


Here is the reason for the two byte offset. "The message is prefixed with a two byte length field which gives the message length, excluding the two byte length field." - RFC 1035 - 4.2.2

This code should use the length field to short-circuit any processing. There is no need to attempt to decode the data when number of bytes is less than the length.

Ok, it will print a debug message "Packet's data is null."

andrewkroh · 2015-12-15T16:44:20Z

I built Packetbeat from your branch and ran some DNS queries over TCP (dig google.com +tcp). My queries are being parsed correctly, but the responses are not.

The events have status: Error and notes: Response packet's data could not be decoded as DNS. I think the cause is the two byte offset.

McStork · 2015-12-15T16:45:43Z

Thanks, I will look into it

McStork · 2015-12-15T17:20:18Z

I couldn't reproduce it. This message is only displayed if you get a Fin or Get with unvalid response DNS data.
All of your dig +tcpcommands fail ? All tests are validating though.

I will keep trying ;)

* return from Parse() if the packet length is <= Offset * check if streams nil in publishDecodeFailureNotes()

andrewkroh · 2015-12-15T18:07:27Z

Try the attached PCAP which contains a single DNS query for google.com over TCP. It should reproduce the Error status.

dns-tcp-google.com.zip

./packetbeat -c etc/packetbeat.dev.yaml -e -v -d "dns,tcp" -I dns-tcp-google.com.pcap -waitstop 10

McStork · 2015-12-16T12:31:21Z

It's good that you could make a split TCP query without DNSSEC.
At the same time it's weird to see that your transaction contains a 1 byte payload in the first response and the rest of the payload in the second response packet ^^.

The offset was not managed how it should have been: * TCP decode Offset is now managed in the function decodeDnsData() * Add a unit test calling the Parse() method on a split query Also: * remove assignements and use pointers to alter DnsStream objects in Parse()

andrewkroh · 2015-12-16T15:20:37Z

packetbeat/protos/dns/dns.go

+			return priv
+		}
+		if dataLength <= DecodeOffset {
+			logp.Debug("dns", EmptyMsg+" addresses %s",


The message is not empty, but it is incomplete and the code will wait for more bytes to complete the message. I think the other protocols log messages for a similar case so maybe take a look at those messages.

DNS - implementing TCP

andrewkroh · 2015-12-16T15:33:09Z

@McStork Thanks for you contribution!

I left a few minor comments about possible enhancements. They are not critical to the feature so I am merging now.

McStork · 2015-12-16T15:37:40Z

@andrewkroh Thank you very much for the reviews. I will open a new PR to address your comments.

urso · 2015-12-16T18:39:07Z

packetbeat/protos/dns/dns.go

+
+	payload := pkt.Payload
+
+	stream := &priv.Data[dir]


uh, pointer of pointer. I'd rather byte the bullet and do the assignment in line 753 instead of having to type (*stream) all over the place.

Check redis code for some cleaned-up solution (using pointers for connection) for cast and append (using streambuf.Buffer though). https://github.com/elastic/beats/blob/master/packetbeat/protos/redis/redis.go#L107

the dnsPrivateData data type is actually the connection context. Consider naming it so

I will do that 😃

andrewkroh reviewed Dec 9, 2015
View reviewed changes

McStork force-pushed the dns-tcp-rebase branch 2 times, most recently from 544a0b7 to 93152d4 Compare December 10, 2015 17:23

McStork force-pushed the dns-tcp-rebase branch from b5d1816 to 9318cb4 Compare December 11, 2015 11:26

McStork force-pushed the dns-tcp-rebase branch from 1031d30 to 4a8d41d Compare December 14, 2015 09:08

tsg added the review label Dec 15, 2015

DNS - implementing Parse for Tcp.

e268f01

* Implement DNS over TCP * Publish Notes when a response fails to decode in Gap and Fin

McStork force-pushed the dns-tcp-rebase branch from 0194483 to e268f01 Compare December 15, 2015 10:02

andrewkroh reviewed Dec 15, 2015
View reviewed changes

Address the review

bef7e71

* return from Parse() if the packet length is <= Offset * check if streams nil in publishDecodeFailureNotes()

McStork force-pushed the dns-tcp-rebase branch from d60694c to bef7e71 Compare December 15, 2015 17:22

Support split payloads

7b8861b

The offset was not managed how it should have been: * TCP decode Offset is now managed in the function decodeDnsData() * Add a unit test calling the Parse() method on a split query Also: * remove assignements and use pointers to alter DnsStream objects in Parse()

McStork force-pushed the dns-tcp-rebase branch from 7bb7369 to 7b8861b Compare December 16, 2015 13:49

andrewkroh reviewed Dec 16, 2015
View reviewed changes

andrewkroh added a commit that referenced this pull request Dec 16, 2015

Merge pull request #486 from McStork/dns-tcp-rebase

9cf0f3d

DNS - implementing TCP

andrewkroh merged commit 9cf0f3d into elastic:master Dec 16, 2015

urso reviewed Dec 16, 2015
View reviewed changes

McStork mentioned this pull request Dec 17, 2015

DNS over TCP clean-up and improvements #554

Merged

McStork deleted the dns-tcp-rebase branch January 6, 2016 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS - implementing TCP #486

DNS - implementing TCP #486

McStork commented Dec 9, 2015

elasticsearch-release commented Dec 9, 2015

andrewkroh Dec 9, 2015

andrewkroh commented Dec 9, 2015

McStork commented Dec 9, 2015

McStork commented Dec 10, 2015

McStork commented Dec 14, 2015

tsg commented Dec 15, 2015

andrewkroh Dec 15, 2015

McStork Dec 15, 2015

andrewkroh commented Dec 15, 2015

McStork commented Dec 15, 2015

McStork commented Dec 15, 2015

andrewkroh commented Dec 15, 2015

McStork commented Dec 16, 2015

andrewkroh Dec 16, 2015

andrewkroh commented Dec 16, 2015

McStork commented Dec 16, 2015

urso Dec 16, 2015

McStork Dec 17, 2015

DNS - implementing TCP #486

DNS - implementing TCP #486

Conversation

McStork commented Dec 9, 2015

elasticsearch-release commented Dec 9, 2015

andrewkroh Dec 9, 2015

Choose a reason for hiding this comment

andrewkroh commented Dec 9, 2015

McStork commented Dec 9, 2015

McStork commented Dec 10, 2015

McStork commented Dec 14, 2015

tsg commented Dec 15, 2015

andrewkroh Dec 15, 2015

Choose a reason for hiding this comment

McStork Dec 15, 2015

Choose a reason for hiding this comment

andrewkroh commented Dec 15, 2015

McStork commented Dec 15, 2015

McStork commented Dec 15, 2015

andrewkroh commented Dec 15, 2015

McStork commented Dec 16, 2015

andrewkroh Dec 16, 2015

Choose a reason for hiding this comment

andrewkroh commented Dec 16, 2015

McStork commented Dec 16, 2015

urso Dec 16, 2015

Choose a reason for hiding this comment

McStork Dec 17, 2015

Choose a reason for hiding this comment